From the medieval European belief that royalty and nobility had blue blood; the elite had enough power and wealth that they could afford to have peasants and the urban poor do their
dirty work for them- since
the aristocrats were able to stay inside and avoid long hours in the fields (and the sunlight), they were often so pale that their blue veins showed under their translucent skin, thus leading people to believe that their blood was blue.
Now it generally refers to people from old money: families who have been wealthy
aristocrats for generations on generations.